Automated and scalable object and feature extraction from imagery

ABSTRACT

Feature extraction of image data using feature extraction modules. The feature extraction modules may be provided in an architecture that allows for modular, decoupled generation and/or operation of the feature extraction modules to generate feature data corresponding to image data. In this regard, the feature extraction modules may communicate with a file system storing image data and feature data by way of a common interface format. Accordingly, regardless of the nature of the execution of the feature extraction module, each feature extraction module may be communicative by way of the common interface format, thereby providing a modular approach that is highly scalable, flexible, and adaptive.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Non-Provisional applicationSer. No. 14/024,399 filed on Sep. 11, 2013, entitled “AUTOMATED ANDSCALABLE OBJECT AND FEATURE EXTRACTION FROM IMAGERY,” the contents ofwhich are incorporated by reference herein as if set forth in full.

BACKGROUND

The use of geospatial imagery has increased in recent years andrepresents a significant tool that may be utilized in a number ofcontexts. As such, high quality geospatial imagery has becomeincreasingly valuable. For example, a variety of different entities(e.g., individuals, governments, corporations, or others) may utilizegeospatial imagery (e.g., satellite imagery). As may be appreciated, theuse of such satellite imagery may vary widely such that geospatialimages may be used for a variety of differing purposes.

With increasingly capable satellites being commissioned and launched,very high resolution (VHR) remotely-sensed multispectral Earth imageryhas become increasingly available and useable. For example, as thenumber of satellite image acquisition systems in operation grows,acquisition ability and flexibility improves. In an example,DigitalGlobe, Inc. of Longmont, Colo. currently operates a number ofsatellites including, IKONOS, GeoEye-1, QuickBird, WorldView 1, andWorldView 2, with an anticipated launch of WorldView 3. Accordingly,around the clock global coverage may be achieved though the satelliteconstellation currently in operation. As such, the DigitalGlobeconstellation of satellites can image the entire Earth's landmass every75 days and may capture over six times the Earth's landmass every yearwith a capacity to collect at least 2.6 million square kilometers ofimagery a day. With selective tasking, DigitalGlobe's satelliteconstellation may quickly and continually collect imagery from targetedlocations to provide near real time feedback on global events or thelike.

Furthermore, the resolution of image acquisition satellites alsocontinues to increase. For instance, currently operated satellites mayhave a maximum spatial resolution of 50 cm (wherein each pixel in theresulting images acquired corresponds with the distance measure of thespatial resolution). Additionally, planned satellite launches mayprovide even greater resolution capabilities with spatial resolutions ashigh as about 30 cm.

In this light, the amount and quality of VHR remotely-sensedmultispectral Earth imagery continues to increase as does the amount andtypes of image data collected. Accordingly, the nature of the VHRremotely-sensed multispectral Earth imagery may facilitate uses beyondsimply providing pixels as image data. For instance, higher level dataprocessing may be applied to the images to, for example, identifyobjects, identify textures, or otherwise extract useful data from theraw image data. In this regard, as the amount of image data that isavailable grows and the nature of the image data acquired changes and isimproved, advanced image data processing and image analytics are neededto keep pace with the advances in image acquisition technology.

SUMMARY

In view of the foregoing, the present disclosure relates to featureextraction from very high resolution (VHR) remotely-sensed Earth imagerysuch as satellite imagery or the like. VHR imagery may comprise remotelysensed imagery with a resolution not less that about 20 m, not less thanabout 10 m, not less than about 5 m, or even not less than about 1 m. Inthis regard and in some embodiments, VHR imagery may make a resolutionas high as 0.50 m or greater. Specifically, the present disclosureincludes architecture and related methodology that facilitates modularand decoupled generation and/or operation of feature extraction modulesthat may be executed in a system to facilitate highly scalable andrapidly evolving techniques for feature extraction. In this regard, thefeature extraction facilitated by the disclosure presented herein mayprovide a base architecture that may support a wide variety ofsolutions, analytics, or other actionable data that may be facilitatedthorough analysis of VHR remotely-sensed Earth imagery. In this regard,the architecture described herein may be used in a variety of specificapplications for analysis of VHR remotely-sensed Earth imageryincluding, for example, data mining and indexing, object detection, landuse analysis, land cover analysis, change identification, or otherapplications without limitations. In at least some embodiments describedherein, the architecture described may also facilitate autonomous imageanalysis that may automatically utilize one or more feature extractionmodules to extract feature data from image data. For example,hierarchical factor extraction models may be provided that perform togenerate feature data based on other faster data generated by otherfeature extraction modules.

Additionally, a number of specific feature extraction modules are alsodetailed in this disclosure. As may be appreciated, analysis of VHRremotely-sensed Earth imagery may present some challenges unique to thenature of the images collected. For instance, while existing imagedetection techniques may make certain assumptions about the orientationof images (e.g., that human figures to be recognized are standing orvertically oriented in an image), these types of assumptions may not beapplicable to VHR remotely-sensed Earth imagery. For instance, theorientation of objects (e.g., as defined by their edges or other salientfeature to be used to identify the object or feature) in VHRremotely-sensed Earth imagery may be provided at any one or more of avariety of orientations or at random. For instance, naturally occurringgeographic phenomena, human made objects, or other types of features,objects, etc. to be identified in VHR remotely-sensed Earth imagery maynot be regularly oriented in the images to be analyzed. Accordingly,previously proposed feature extraction techniques that make certainassumptions about the orientation of an image or that are specificallytailored to analyze an image with a given orientation may not be suitedto analysis of VHR remotely-sensed Earth imagery. In this regard, atleast some of the feature extraction modules described herein may berotationally invariant.

A first aspect includes a method for extracting feature data from veryhigh resolution (VHR) remotely-sensed multispectral Earth imagery. Themethod includes storing VHR remotely-sensed multispectral Earth imagedata in a file system and providing a plurality of feature extractionmodules. Each respective feature extraction module is operable toanalyze the image data to generate feature data regarding the imagedata. As such, the method includes accessing the VHR remotely-sensedmultispectral Earth image data in the file system with one or more ofthe plurality of feature extraction modules, wherein the one or morefeature extraction modules each access the file system using a commoninterface format. The method further includes executing the one or morefeature extraction modules to generate feature data corresponding to theVHR remotely-sensed multispectral Earth image data and receiving thefeature data at the file system from the one or more feature extractionmodules. The feature data is received from each of the one or morefeature extraction modules using the common interface format.

A number of feature refinements and additional features are applicableto the first aspect. These feature refinements and additional featuresmay be used individually or in any combination. As such, each of thefollowing features that will be discussed may be, but are not requiredto be, used with any other feature or combination of features of thefirst aspect.

For example, in an embodiment, the VHR remotely-sensed multispectralEarth image data may include spectral band data corresponding to atleast 8 multispectral bands. In this regard, the multispectral bands maycollectively range from at least about 300 nanometers in wavelength toat least about 1100 nanometers in wavelength. Additionally, the imagedata may further include a plurality of short-wavelength infrared (SWIR)bands. The plurality of SWIR bands may range from at least about 1100nanometers in wavelength to at least about 2400 nanometers inwavelength.

In an embodiment, the feature information received from the one or morefeature extraction modules may be stored in corresponding relation tothe portion of the VHR remotely-sensed multispectral Earth image datafrom which the feature data was generated. Furthermore, in anembodiment, at least two of the plurality of feature extraction modulescomprises different programming languages. For instance, the pluralityof feature extraction modules may be independently developed and eachcommunicate with the file system using the common interface format.Accordingly, the common interface format comprises a feature identifierand an image identifier. The feature identifier may correspond withparticular feature data generated by a given feature extraction module.Additionally, the image identifier comprises an indication of a portionof the image for which feature data is to be generated.

In an embodiment, the method may also include determining a subset ofthe plurality of feature extraction modules required to generate featuredata, wherein first feature data generated from the execution of a firstfeature extraction module is used during the executing of a secondfeature extraction module to generate second feature data. Thedetermining may also include ordering the execution of the subset offeature extraction modules. Also, the method may include distributing aninitialization command to each of the subset of the plurality of featureextraction modules, wherein the initialization command comprises afeature identifier and an image identifier according to the commoninterface format. In an embodiment, at least one of the featureextraction modules generates feature data prior to receiving a requestfor the feature data from an image processing module. Additionally oralternatively, at least one of the feature extraction modules maygenerate feature data on demand in response to a request for the featuredata from an image processing module. The determining may be at leastpartially based on a feature of interest to be identified from the imagedata as requested from an imaging processing module.

The feature extraction modules may generally be any appropriate modulethat is operable to generate feature data. For example, the featureextraction modules may include a fractal dimension feature extractionmodule, a total variational feature extraction module, a rotationallyinvariant histogram of gradients (HOG) feature extraction module, aGabor wavelet feature extraction module, a clustering histogram featureextraction module, or any other appropriate feature extraction module.

Accordingly, the fractal dimension feature extraction module may beoperative to generate feature data by identifying an image of apredetermined size of pixels, dividing the image into a plurality ofabutting square windows having a side dimension less than thepredetermined size of the image, and constructing for each plurality ofabutting square windows a column having a height as represented inincrements of the side dimension of the abutting window. The height ofthe column may be determined such that the ratio of the predeterminedsize of the image to the side dimension of the abutting windows equalsthe ratio of a maximum pixel value for a given multispectral band of theimage to the height. The squared side dimension and the height mayrepresent a plurality of boxes whose volume is defined by increments ofthe side dimension. In this regard, the fractal dimension featureextraction module may determine, for each abutting window a box from theplurality of boxes in which a maximum pixel value and a minimum pixelvalue occur and may calculate, for each of the plurality of abuttingwindows, the number of boxes separating the box in which the maximumpixel value is disposed and the box in which the minimum pixel value isdisposed. The module may also sum the number of boxes separating maximumand minimum pixel levels over all of the plurality of abutting windowsto generate a summed value. In turn, the module may repeat each of theconstructing, determining, calculating, and summing operations for allpossible values of the side dimension of the plurality of abuttingwindows such that each side dimension is less than or equal to half ofthe predetermined size of the image and at least greater than threepixels in length. Additionally, the module may plot each summed valuegenerated in the repeating step to determine a slope of the summedvalues, wherein the slope comprises the fractal dimension of the image.

In an embodiment, the image of the predetermined size is a subsetportion of a larger image. In turn, the fractal dimension may becalculated for each subset portion of the larger image is used togenerate a fractal map comprising an aggregated mapping of the fractalvalues of each subset portion over the larger image. As such, a fractalmap may be generated for each spectral band of a multispectral image.The fractal dimension may be at least partially dependent upon thespectral band, wherein each fractal map is independently generated ofother fractal maps of the other spectral bands of the plurality ofspectral bands.

In another embodiment, at least one of the feature extraction modulesmay include a total variational (TV) feature extraction module. The TVfeature extraction module may include a multispectral filtering moduleusing a plurality of spectral bands of an image to generate a filteredimage corresponding to the image. The number of spectral bands used inthe filtering operation may be at least 8 multispectral bands. Thefiltering operation may include minimizing a function including at leasta first term representing the difference between the original image andthe filtered image and at least a second term representing the spatialhomogeneity of the filtered image. In this regard, the first term andthe second term may include a multivalued vector for each pixelrepresentative of the image or filtered image, respectively, wherein themultivalued vector values include a component for each row, column, andmultispectral band in the image. At least one of the first term or thesecond term include an L1 norm. Additionally, the second term isweighted by a smoothing factor. The minimization of the function may besolved using a Split Bregman technique. The Split Bregman technique mayinclude Gauss-Seidel updates. In this regard, the minimization is solvedby execution on a graphics processing unit (GPU).

In an embodiment, at least one of the feature extraction modules mayinclude a rotational invariant histogram of gradients (HOG) featureextraction module. The rotational invariant HOG feature extractionmodule may be operative to generate feature data by establishing aplurality of angular bins for an image in a gradient domain,histogramming gradients from the image with respect to the plurality ofangular bins, and selecting based on the histogramming the angular binwith the largest histogram value to define a primary direction.Additionally, the module may set a plurality of pooling windows withrelative offsets from a pixel of interest and define a gradientorientation window encompassing the plurality of pooling windows. Themodule may configure the gradient orientation window to align theconfiguration of the gradient orientation window relative to the primarydirection. Also, the module may shift the positions of the poolingwindows relative to the gradient orientation window based on theconfiguring. The module may also rotate the internal orientation of thepooling windows based on the configuring.

In an embodiment, the shifting may include shifting the pooling windowsrelative to the gradient orientation window by a number of binsdetermined in the selecting step. The rotating may also include shiftingthe internal orientation of the pooling windows by a number of binsdetermined by the rotating step. Accordingly, a HOG is calculatedrelative to each of the shifted and rotated pooling windows. Thecalculating may include using an integral image to calculate the HOGfeature for each pooling window.

In an embodiment, at least one of the feature extraction modules mayinclude a Gabor wavelet feature extraction module. The Gabor waveletfeature extraction modules may be operative to generate feature data byapplying a plurality of Gabor wavelet functions to an image, whereineach of the plurality of Gabor wavelet function includes a differentorientation and computing at least one feature based on at least two ofthe plurality of Gabor wavelet functions. For example, the at least onefeature may include at least one of a sum of the absolute values of afirst Gabor wavelet function of the plurality of Gabor wavelet functionsand a second Gabor wavelet function of the plurality of Gabor waveletfunctions, a sum of the magnitude of values of the product of eachperpendicularly oriented Gabor wavelet function of the plurality ofGabor wavelet functions, or the difference of the maximum and minimumvalues of magnitude of all of the plurality of Gabor wavelet functionsfor all image data. As such, for each orientation of the plurality ofGabor wavelet functions, a plurality of different Gabor waveletfunctions with a different scale are calculated. The different scalesmay include varied values for at least one of a wavelength of the Gaborwavelet function or an envelope of the Gabor wavelet function. Each ofthe plurality of Gabor wavelet functions may be applied to each pixel ofeach multispectral band of an image comprising a plurality ofmultispectral bands. At least one feature may be determinedindependently for each of the plurality of bands. Additionally oralternatively, the at least one feature may be determined with respectto all of the plurality of bands.

In an embodiment, at least one of the feature extraction modules mayinclude a clustering histogram feature extraction module for extractionof features based on clustered and classified spectra values. Theclustering histogram feature extraction module may be operative togenerate feature data by assigning each pixel of an image into one of aplurality of a predefined classes and identifying a kernel surroundingeach pixel containing a plurality of others pixels from the image,wherein each of the plurality of other pixels have been assigned intoone of the plurality of predefined classes. Furthermore, the module maybuild a histogram for each pixel based on the number of other pixels ineach of the plurality of predefined classes relative to the pixel forwhich the histogram is built. The plurality of predefined classes maycorrespond to identified land classifications based on at least one ofland use or land class. The pixels may be assigned into a predefinedclass based on the radiometric properties of a pixel used in aclustering technique. The clustering technique may include at least oneof k-means clustering, a Euclidean distance approach, or a Mahalanobisdistance approach.

A second aspect includes a system for generating feature data from veryhigh resolution (VHR) remotely-sensed multispectral Earth imagery. Thesystem includes a data store comprising VHR remotely-sensedmultispectral Earth image data in a file system, and a plurality offeature extraction modules, wherein each respective feature extractionmodule is operable to analyze the image data to generate feature dataregarding the image data. The one or more feature extraction moduleseach access the file system using a common interface format to accessthe image data and store feature data.

A number of feature refinements and additional features are applicableto the second aspect. These feature refinements and additional featuresmay be used individually or in any combination. As such, each of theforegoing features discussed in relation to the first aspect may be, butare not required to be, used with any other feature or combination offeatures of the second aspect.

A third aspect includes a method for generating feature data from imagedata using a plurality of hierarchical feature extraction modules. Themethod includes providing a plurality of feature extraction modules.Each respective feature extraction module is operable to generatecorresponding feature data for image data, wherein the plurality offeature extraction modules comprises at least a first feature extractionmodules operable to generate first feature data and a second featureextraction module operable to generate second feature data. The methodmay include first generating the first feature data with the firstfeature extraction module and communicating the first feature data tothe second feature extraction module using a common interface format. Inturn, the method may include second generating the second feature dataat least in part based on the first feature data.

A number of feature refinements and additional features are applicableto the third aspect. These feature refinements and additional featuresmay be used individually or in any combination. As such, each of theforegoing features discussed in relation to the first or second aspectmay be, but are not required to be, used with any other feature orcombination of features of the third aspect.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying drawings illustrate several embodiments of the presentdisclosure and, together with the description, serve to explain theprinciples according to the embodiments. One skilled in the art willrecognize that the particular embodiments illustrated in the drawingsare merely exemplary, and are not intended to limit the scope of thepresent disclosure.

FIG. 1 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in an embodiment.

FIG. 2 is a block diagram illustrating an exemplary logical architecturefor a client device, according to an embodiment.

FIG. 3 is a block diagram showing an exemplary architectural arrangementof clients, servers, and external services, according to an embodiment.

FIG. 4 is a schematic view of an embodiment of a feature extractionsystem.

FIG. 5 is a graphical representation of an embodiment of a feature stackof feature data that may be generated for corresponding image data usingan embodiment of a feature extraction system.

FIG. 6 depicts an embodiment of a graphical representation of a methodfor determining a fractal dimension of an image.

FIG. 7 depicts an embodiment of a plurality of bins for use in ahistogram of gradients feature extraction technique.

FIGS. 8A and 8B depict an embodiment of a graphical representation of arotationally invariant histogram of gradients technique applied to animage.

FIG. 9 depicts an embodiment of interaction between various featureextraction modules by way of a common interface format.

DETAILED DESCRIPTION

The following description is not intended to limit the invention to theforms disclosed herein. Consequently, variations and modificationscommensurate with the following teachings, skill and knowledge of therelevant art, are within the scope of the present invention. Theembodiments described herein are further intended to explain modes knownof practicing the invention and to enable others skilled in the art toutilize the invention in such, or other embodiments and with variousmodifications required by the particular applications(s) or use(s) ofthe present invention.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beunderstood that these are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the inventions may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. In general,embodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the inventions, and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the particular inventions. Accordingly,those skilled in the art will recognize that one or more of theinventions may be practiced with various modifications and alterations.Particular features of one or more of the inventions may be describedwith reference to one or more particular embodiments or figures thatform a part of the present disclosure, and in which are shown, by way ofillustration, specific embodiments of one or more of the inventions. Itshould be understood, however, that such features are not limited tousage in the one or more particular embodiments or figures withreference to which they are described. The present disclosure is neithera literal description of all embodiments of one or more of theinventions nor a listing of features of one or more of the inventionsthat must be present in all embodiments.

Devices that are described as being in communication with each otherneed not be in continuous communication with each other, unlessexpressly specified otherwise. In addition, devices that are describedas being in communication with each other may communicate directly orindirectly through one or more intermediaries, logical or physical. Adescription of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions.

Similarly, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may generally be configured to work in alternate orders,unless specifically stated to the contrary. In other words, any sequenceor order of steps that may be described in this patent application doesnot, in and of itself, indicate a requirement that the steps beperformed in that order. The steps of described processes may beperformed in any order practical. Further, some steps may be performedsimultaneously despite being described or implied as occurringnon-simultaneously (e.g., because one step is described after the otherstep). Moreover, the illustration of a process by its depiction in adrawing does not imply that the illustrated process is exclusive ofother variations and modifications thereto, does not imply that theillustrated process or any of its steps are necessary to one or more ofthe invention(s), and does not imply that the illustrated process ispreferred. Also, steps are generally described once per embodiment, butthis does not mean they must occur once, or that they may only occuronce each time a process, method, or algorithm is carried out orexecuted. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple instantiations of a mechanism unless noted otherwise.Process descriptions or blocks in figures should be understood asrepresenting modules, segments, or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process. Alternate implementations are included withinthe scope of embodiments of the present invention in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

As used herein, a “feature” may correspond to an individual measurableheuristic property of a phenomenon being observed (e.g., a value relatedto image data). In this regard, features or feature data may beextracted from image data (e.g., by performing calculations and/ortransformations of image data). As used herein, a “pixel-level feature”is a feature at a base level of classification. For example, pixel-levelfeatures may include textural properties, mathematical transformsapplied to pixel data, a land-cover classification, or some otherappropriate property description specific to individual pixels. As usedherein, a “region-level feature” is a feature at a higher level ofclassification. For example, region-level feature observations may berelated to morphology, i.e., may have shape properties (such as, area,perimeter, compactness, elongation, eccentricity, etc.), spatialrelationships (such as, arrangement, distance, etc.), objectclassifications (for example, a school, paring lot, swimming pool,plane, shopping mall, etc.), and the like. As used herein, a“scene-level feature” is a feature that may aggregate statistics onlower level features (e.g., region-level features or pixel-levelfeatures), such as, percentage land cover (for example, 25% deciduousforest, 10% water, etc.), aggregate object counts (for example, 10schools, 35 parking lots, etc.), other descriptive classifications (forexample, desert, city, rainforest, etc.)

As used herein, a “graphics processing unit (GPU)” is a specializedelectronic circuit known in the art that is designed to rapidlymanipulate and alter memory to accelerate the creation of images in aframe buffer intended for output to a display or for other computationalpurposes. As used herein, a “Compute Unified Device Architecture (CUDA)”is a parallel computing platform and programming model known in the artcreated by NVIDIA™ and implemented by the graphics processing units(GPUs) that they produce that gives developers access to a virtualinstruction set and memory of the parallel computational elements inCUDA GPUs.

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be disclosedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, and the like), aconsumer electronic device, a music player, or any other suitableelectronic device, router, switch, or the like, or any combinationthereof. In at least some embodiments, at least some of the features orfunctionalities of the various embodiments disclosed herein may beimplemented in one or more virtualized computing environments (e.g.,network computing clouds, virtual machines hosted on one or morephysical computing machines, or the like).

Referring now to FIG. 1, there is shown a block diagram depicting anexemplary computing device 100 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 100 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 100 may be adaptedto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork, a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 100 includes one or more centralprocessing units (CPU) 102, one or more interfaces 110, and one or morebusses 106 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 102may be responsible for implementing specific functions associated withthe functions of a specifically configured computing device or machine.For example, in at least one embodiment, a computing device 100 may beconfigured or designed to function as a server system utilizing CPU 102,local memory 101 and/or remote memory 120, and interface(s) 110. In atleast one embodiment, CPU 102 may be caused to perform one or more ofthe different types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 102 may include one or more processors 103 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 103 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 100. In a specificembodiment, a local memory 101 (such as non-volatile random accessmemory (RAM) and/or read-only memory (ROM), including for example one ormore levels of cached memory) may also form part of CPU 102. However,there are many different ways in which memory may be coupled to system100. Memory 101 may be used for a variety of purposes such as, forexample, caching and/or storing data, programming instructions, and thelike. As used herein, the term “processor” is not limited merely tothose integrated circuits referred to in the art as a processor, amobile processor, or a microprocessor, but broadly refers to amicrocontroller, a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 110 are provided as network interfacecards (NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 110 may forexample support other peripherals used with computing device 100. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, Firewire™, PCI, parallel, radio frequency (RF),Bluetooth™ near-field communications (e.g., using near-field magnetics),802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces,Gigabit Ethernet interfaces, asynchronous transfer mode (ATM)interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale(POS) interfaces, fiber data distributed interfaces (FDDIs), and thelike. Generally, such interfaces 110 may include ports appropriate forcommunication with appropriate media. In some cases, they may alsoinclude an independent processor and, in some in stances, volatileand/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 1 illustrates one specificarchitecture for a computing device 100 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 103 may be used, and such processors 103may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 103 handlescommunications as well as routing computations, while in otherembodiments a separate dedicated communications processor may beprovided. In various embodiments, different types of features orfunctionalities may be implemented in a system according to theinvention that includes a client device (such as a tablet device orsmartphone running client software) and server systems (such as a serversystem described in more detail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 120 and local memory 101) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 120 or memories101, 120 may also be configured to store data structures, configurationdata, encryption data, historical system operations information, or anyother specific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory, solid state drives, memristormemory, random access memory (RAM), and the like. Examples of programinstructions include both object code, such as may be produced by acompiler, machine code, such as may be produced by an assembler or alinker, byte code, such as may be generated by for example a Java™compiler and may be executed using a Java virtual machine (JVM) orequivalent, or files containing higher level code that may be executedby the computer using an interpreter (for example, scripts written inPython, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 2,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 200 includes processors210 that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 230. Processors 210 may carry out computinginstructions under control of an operating system 220 such as, forexample, a version of Microsoft's Windows™ operating system, Apple's MacOS/X or iOS operating systems, some variety of the Linux operatingsystem, Google's Android™ operating system, or the like. In many cases,one or more shared services 225 may be operable in system 200, and maybe useful for providing common services to client applications 230.Services 225 may for example be Windows™ services, user-space commonservices in a Linux environment, or any other type of common servicearchitecture used with operating system 210. Input devices 270 may be ofany type suitable for receiving user input, including for example akeyboard, touchscreen, microphone (for example, for voice input), mouse,touchpad, trackball, or any combination thereof. Output devices 260 maybe of any type suitable for providing output to one or more users,whether remote or local to system 200, and may include for example oneor more screens for visual output, speakers, printers, or anycombination thereof. Memory 240 may be random-access memory having anystructure and architecture known in the art, for use by processors 210,for example to run software. Storage devices 250 may be any magnetic,optical, mechanical, memristor, or electrical storage device for storageof data in digital form. Examples of storage devices 250 include flashmemory, magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 3, there is shown a blockdiagram depicting an exemplary architecture for implementing at least aportion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 330 may be provided. Each client 330 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 200 such as that illustrated in FIG. 2. In addition,any number of servers 320 may be provided for handling requests receivedfrom one or more clients 330. Clients 330 and servers 320 maycommunicate with one another via one or more electronic networks 310,which may be in various embodiments any of the Internet, a wide areanetwork, a mobile telephony network, a wireless network (such as WiFi,Wimax, and so forth), or a local area network (or indeed any networktopology known in the art; the invention does not prefer any one networktopology over any other). Networks 310 may be implemented using anyknown network protocols, including for example wired and/or wirelessprotocols.

In addition, in some embodiments, servers 320 may call external services370 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 370 may take place, for example, via one or morenetworks 310. In various embodiments, external services 370 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 230 are implemented on a smartphone or other electronicdevice, client applications 230 may obtain information stored in aserver system 320 in the cloud or on an external service 370 deployed onone or more of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 330 or servers 320 (orboth) may make use of one or more specialized services or appliancesthat may be deployed locally or remotely across one or more networks310. For example, one or more databases 340 may be used or referred toby one or more embodiments of the invention. It should be understood byone having ordinary skill in the art that databases 340 may be arrangedin a wide variety of architectures and using a wide variety of dataaccess and manipulation means. For example, in various embodiments oneor more databases 340 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop Cassandra, Google BigTable, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 360 and configuration systems 350. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 360 orconfiguration system 350 or approach is specifically required by thedescription of any specific embodiment.

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

FIG. 4 is a block diagram illustrating an exemplary feature extractionsystem 400 that may be used for a variety of applications whereextracted feature data from image data may assist in image analysisincluding, for example, modular image mining and search, landclassification, object detection, change detection, or any otherapplication in which extracted features of image data may be useful. Asillustrated, system 400 may comprise a plurality of software or hardwareelements that may be variably interconnected, and it should beappreciated that such components may be variable in implementation; thatis, components may be software-based and stored and operating onseparate or joined hardware systems interchangeably (for example byrunning multiple software systems on a single computer server), or maybe physically-distinct hardware such as dedicated computing hardware forspecific elements as illustrated. It should be further appreciated thatconnections between components as illustrated may be direct physicalconnections (such as via cables or internal software connectivity withina hardware system operating multiple components as described above) orthey may be remote data connections such as communication over theInternet or another communications network. In this manner, it should beappreciated that the constructions and arrangement illustrated isexemplary, and a variety of arrangements may be possible, and it shouldbe further appreciated that the quantity and arrangement of componentsis exemplary and alternate or additional components may be utilized. Inthis manner it can be appreciated that the nature of the system 400 isflexible and scalable, i.e. it may be arranged as suitable for varyingparticular implementations as may be appropriate (such as a small-scaleimplementation utilizing software elements operating within a personalcomputer for individual use, or a large-scale implementation utilizingseparate computer servers for operation utilizing greater computingpower or storage space, as may be desirable for an enterprise orcloud-based service).

As illustrated, image data 411 may be stored in a data store 410 orsimilar data storage (e.g., a database or the like). Image data 411 maybe of variable nature such as simple stored images or more detailedinformation or “metadata” stored along with images such as image tags orother identifying data (such as, as is common in the art, datapertaining to when, where, or how an image was captured, such as mightbe recorded by an image capture device 412 when taking a snapshot).Furthermore, image data 411 need not be an existing cache of storeddata, and could be a quantity of “live” data being accessed by system400 for operation such as capturing images via any appropriateimage-capture devices 412 connected to system 400 such as cameras 413,image capture-capable electronic devices such as tablet computingdevices or smartphones 414, or aerial or satellite imaging systems 415,in order that captured images may be indexed directly without firstbeing stored.

It should be appreciated that the use of a database as the data store410 is exemplary, and that image storage may be accomplished via avariety of means common in the art, such as a single physical storagemedium (such as magnetic, optical, solid-state, or other storage devicescommon in the art), or a distributed storage system (such as adistributed file storage system as may be utilized in computingclusters), and further that such storage may be local (i.e., stored andoperating alongside other components of system 400 such as a server'shard disk drive or a connected storage system) or remote (such as aremotely-connected and accessible storage device or a cloud-basedstorage service), and that connection may be accomplished via eitherphysical or remote means, as described previously. In this manner itshould be appreciated that image storage is highly variable, furtherenabling a flexible and scalable design that may be readily adapted to avariety of storage means as may be appropriate, without necessitatingthe manipulation of existing data.

The image data 411 may undergo analysis by one or more featureextraction modules 450 shown in FIG. 4. In this regard, featureextraction modules 450 may be operable to access the image data 411 andanalyze the image data 411 to generate feature data 420. For example,feature data 420 to be extracted from the images may comprise“pixel-level” features where individual pixels within an image may bescanned and identified such as to determine textural information (suchas may be appropriate in determining whether a photographed object ismade of metal or wood, or for determining terrain in a satellite imageof an area). In this regard, pixel-level feature extractors may includespectral analysis performed relative to pixels, mathematical transformsof spectral data concerning pixels, or other processing on apixel-level. In turn, it may be appreciated that pixel-level featuresmay also be useful in feature extraction modules for determining“segmentation” of image elements (such as identifying the separation ofindividual objects, like separating buildings in a photograph of askyline or the edge of a forest in a satellite image).

In a further “region-level” feature extraction operation, an image maybe processed for mid-level granular information, optionally usingpixel-level information from a previous feature extraction operation.For example, using previously extracted segmentation information, aregion-level indexing operation might identify the nature of objects orregions identified previously (continuing from the previous example,identifying that one region of a satellite image is a forest whileanother is an urban sprawl, detecting objects from an image such ascars, parking lots, buildings, or the like). In this regard and as willbe described in greater detail below, feature extraction modules 450 maybe at least partially hierarchical such that a feature extraction module450 may generate feature data 420 based on other feature data 420generated from another feature extraction module 450.

In a further “scene-level” feature extraction operation, an image may beprocessed for the highest level of granular information, such as (again,continuing previous examples) analyzing what percentage of a satelliteimage is forested, or counting the number of objects in an image.

In this regard, the extracted feature data 420 may be provided as afeature stack 500, graphically illustrated in FIG. 5. With furtherreference to FIG. 5, it may be appreciated that for each portion ofimage data 411, a plurality of layers of information or feature data 420related to or comprising subcomponents of the image data 411 may beprovided in the form of a feature stack 500. In this regard, the featurestack 500 may include image data 411 (e.g., spectral band data 610 foreach pixel) and one or more layers generated by way of a featureextraction module 450. For instance, the spectral band data 510 mayinclude pixel value (e.g., a gray level value for each pixel for eachspectral band provided). In this regard, as shown in FIG. 5, eightspectral bands are provided corresponding to, for example, spectral datacollected from the eight spectral detectors of the WorldView 2 satelliteimaging system 415 operated by DigitalGlobe, Inc. However, additional orfewer numbers of spectral bands may be provided in the spectral banddata 510. The image data 411 may be orthorectified and/oratmospherically compensated satellite imagery. Other image processingcommon to remotely-sensed imagery may also be provided withoutlimitation. In this regard, feature data 420 may be more portable as theimage data 411 may be atmospherically compensated, thus providing moreconsistency between image data 411 for different geographies oratmospheric conditions.

In any regard, a visible image may be generated that may includecomponents from one or more of the layers of the feature stack 500. Forinstance, in one embodiment, data from the spectral band data 510corresponding to wavelengths visible to the human eye, such as forexample, the red, green, and blue bands may be combined to produce animage representative of the image as visible to the human eye (e.g., anatural color image). This layer may be stored as a spectral featurelayer 520 as it may comprise a function (i.e., an extracted feature)that is based on one or more of the spectral band data 510. Furthermore,visible images may be generated based on gray level values from spectralband data 510 not visible to the human eye to generate visiblerepresentations of such bands (e.g., alone or in combination with otherhuman visible and/or nonhuman visible bands) to generate false colorimages. In this regard, the spectral feature layers 520 of the featurestack 500 may include, for example, layers that include data based onspectral band data 510 of the image data 411 including, for example,combinations thereof.

Accordingly, the extraction of a limited number of bands (e.g., the red,green and blue bands or “RGB” bands to construct an RGB natural colorimage) may allow for reduced computational overhead when presenting theimage data to the user. That is, rather than loading and/or attemptingto display all values of all bands available for an image, the RGB bandsmay be used to present an image that represents the human-visibleportion of the spectrum. Accordingly, where the user views ormanipulates the image the use of the RGB image may be more efficientlyloaded and/or manipulated with less computational overhead.

The spectral feature layers 520 may also include ratios of gray valuesfor one or more bands for each pixel based on the spectral band data510. For example, in one ratio of spectral bands that may be generatedas a layer includes:

$\begin{matrix}\frac{\left( {{{Band}\mspace{14mu} A_{i}} - {{Band}\mspace{14mu} B_{i}}} \right)}{\left( {{{Band}\mspace{14mu} A_{i}} + {{Band}\mspace{14mu} B_{i}}} \right)} & {{Equation}\mspace{14mu} 1}\end{matrix}$

where Band A_(i) represents a pixel value from a first spectral band inthe spectral band data 510 for a given pixel i, and Band B_(i)represents a pixel value from a second spectral band in the spectralband data 510 for the given pixel i.

The feature extraction modules 450 may also be operable to generate datalayers that include feature data 420 for a given portion of image data411. For example, textural feature layers 530 and/or morphologicalfeature layers 540 may also be generated for a portion of image data 411that form a portion of the feature stack 500 as will be described ingreater detail below.

In any regard, it may be appreciated that the layers in the featurestack 500 may each include corresponding pixel feature data for a pixelfrom the image data 411. For example, in FIG. 5, the image data 411 mayinclude a nine pixel by nine pixel portion of image data 411. Ahighlighted pixel 550 has been denoted with an “x.” Values for pixels550 a-550 f are also provided in various feature data layers thatcorrespond to pixel 550 and also denoted in the feature data layers ofthe feature stack 500. These additional corresponding pixels 550 a-550 fare each provided in a corresponding position in a respective featuredata layer to pixel 550. That is, for each feature data layer, acorresponding nine by nine pixel map may be provided with valuescorresponding to the feature data for each one of the pixels of theimage data 411. It may be appreciated that the resulting feature datalayer generated by a feature extraction module 450 may, but need not,correspond to the size of the portion of image data 411 used to generatethe layer. That is, the resulting data layer may be a larger or smallerarray than the image data 411. Furthermore, each pixel in a feature datalayer may be a vector value or other type of output corresponding to aportion of the image data 411. However, in the example depicted in FIG.5, for example, pixels 550 a and 550 b may include values correspondingto spectral features extracted from pixel 550 (e.g., from one or more ofthe spectral band data 510 corresponding to pixel value 550). Pixels 550c and 550 d may include values corresponding to textural featuresextracted based on a feature extraction operation carried out relativeto pixel 550. Pixels 550 e and 550 f may include values corresponding tomorphological features extracted based on a feature extraction operationcarried out relative to pixel 550. In any regard, it may be appreciatedthat for a given pixel 550, corresponding feature values (i.e., pixels550 a-550 f) may be provided that correspond to the pixel 550. As such,the values for pixels 550 a-550 f may be associated with pixel 550 suchthat the values for each of the pixels may be associated.

It may further be appreciated that in the case of VHR remotely-sensedmultispectral Earth imagery, each pixel 550 may be correlated to acorresponding geographic position. As such, for each pixel 550, ageographic identifier may be provided that is indicative of thegeographic location corresponding to a given pixel 550.

Textural feature layers 530 may be generated using feature extractionmodules 450 including, for example, Gabor filter feature data, histogramof oriented gradients (HOG) features, or any appropriate other texturalfeatures. Morphological feature layers 540 may be generated usingfeature extraction modules 450 including, for example, identification ofhuman buildup or the like (e.g., as discussed in U.S. patent applicationSer. No. 14/013,904 entitled AUTOMATIC EXTRACTION OF BUILT-UP FOOTPRINTSFROM HIGH RESOLUTION OVERHEAD IMAGERY THROUGH MANIPULATION OF ALPHATREEDATA STRUCTURES filed on Aug. 29, 2013, the entirety of which isincorporated by reference). As may be appreciated from FIG. 5, there maybe any number of spectral feature layers 520, textural feature layers530, and/or morphological feature layers 540 provided in the featurestack 500.

Specifically, the present disclosure may facilitate feature extractionof features from one image data, wherein the feature extraction system400 is autonomous and/or highly scalable. For example, in the specificcontext of feature extraction from very high resolution (VHR)remotely-sensed multispectral Earth imagery, the number of features thatmay be generated from such imagery may be numerous. Furthermore, featureextraction modules 450 may be operable to build upon previouslyextracted feature data 420 or such that sequential execution of aplurality of feature extraction modules 450 is carried out to generatefeature data 420. In this regard, the environment in which such featureextraction modules 450 are provided may be advantageously structured toallow for independent development of feature extraction modules 450wherein the feature extraction modules 450 are easily inter-operablewith one another and a file system 430 used for storing the image data411 and feature data 420.

Accordingly, the feature extraction system 400 may be provided thatfacilitates an autonomous, modular, and decoupled approach to a featureextraction operation to be performed on imagery. In FIG. 4, a filesystem 430 may be provided that may be used to store image data 411therein. The image data 411 and/or feature data 420 may be accessible bythe one or more feature extraction modules 450. Each one of theplurality of feature extraction modules 450 may be operable to extractfrom the image data 411 or previously generated feature data 420 one ormore portions of feature data 420 for which the module 450 isspecifically adapted. Examples of feature extraction modules 450 areprovided in greater detail below.

In any regard, each feature extraction module 450 may be operable togenerate feature data 420. The feature data 420 (e.g., a data layer of afeature stack 500) may be provided from the feature extraction module450 and stored using the file system 430 of the data store 410. In thisregard, feature data 420 that is extracted from the image data 411 maybe stored in the file system 430 in corresponding relation to the imagedata 411 from which the feature data 420 was generated. Thecorrespondence may be provided as a correlation between pixels of theimage data 411 and a corresponding value in a data layer of the featurestack 500 (e.g., as described above in relation to the feature stack500) or, in the case of geospatial imagery, as a correlation betweengeospatial values (e.g., latitude and longitude values or the like).

As described above, given the potential for many types of features to beof interest, many different feature extraction modules 450 may beprovided that each correspond to different feature data 420 to beextracted from the image data 411. Given the potential variety andpotentially differing objectives of the feature extraction modules 450,a modular, decoupled approach to the architecture and use of featureextraction modules 450 in the feature extraction system 400 used may bepreferred to facilitate the wide variety of feature data that may begenerated by differing feature extraction modules 450. As such, eachfeature extraction module 450 may be in communication with the filesystem 430 by way of a common interface format 440. In this regard, thecommon interface format 440 may provide a predefined format forcommunication between the feature extraction modules 450 and the filesystem 430 for both retrieval of image data 411 and communication offeature data 420.

In this regard, the feature extraction modules 450 may be independentlydeveloped and/or executed. That is, the feature extraction modules 450may be built with differing architecture, programming languages, and/orprogramming paradigms such that the design of the feature extractionmodules 450 may vary. However, given the use of a predeterminedinterface format 440, intercommunication between the feature extractionmodules 450 and the file system 420 may be facilitated regardless of thenature of the feature extraction module 450. Accordingly, featureextraction modules 450 may be independent and modular with communicationfacilitated by way of the common interface format 440. Thus, the featureextraction system 400 may be highly adaptable and scalable. Forinstance, feature extraction modules 450 may be independently developed,yet retain interoperability though the common interface format 440. Inthis regard and as stated above, feature extraction modules 450 maycomprise different programming languages or the like. Furthermore, aseach feature extraction module 450 may utilize the predeterminedinterface format 440 for input and output communications, exchange ofdata between feature extraction modules 450 may also be according to thecommon interface format 440. In this regard, inter-module communicationsmay be facilitated so as to allow for sequential execution of aplurality of feature extraction modules 450 as will be described ingreater detail below. Accordingly, in addition to accessing image data411, a feature extraction module 450 may also be operable to accessfeature data 420 from other feature extraction modules 450 by way of thecommon interface format 440.

With further reference to FIG. 9, one embodiment of a structure 900 forinteraction of feature extraction modules 450 is shown. As may beappreciated, use of a common interface format 440 may allow for a moreefficient and user friendly approach to facilitating generation offeature data 420 by way of feature extraction modules 450. For instance,as shown in FIG. 9, a request 910 for a particular feature (e.g.,Feature 1) corresponding to a particular portion of an image may beprovided. In this regard, the common interface format 440 may include afeature identifier 442 and an image identifier 444. Accordingly, as maybe appreciated, feature data 420 generated by a given module may beidentified by a corresponding feature identifier 442. As such, using theexample shown in FIG. 9, request 910 may include a feature identifier442 corresponding with “Feature 1” produced by feature extraction module1 450 a. As such, the request 910 may be formatted according to thecommon interface format 440 such that the feature identifier 442 may beused to identify which module 450 should be called for execution from aplurality of modules 450 that may be provided with the system 400.

Additionally, the request 910 may include an image identifier 444. Asshown, the image identifier 444 may be provided in the common interfaceformat 440. Specifically, the image identifier 444 may be a function ofa given image i and particular pixel ranges within the image i such asfor example a horizontal range denoted in FIG. 9 as x and a verticalrange denoted as y. It will be appreciated that x and/or y may be arange of pixels to define a region within image i for which the featuredata 420 is requested. In this regard, upon provision of the featureidentifier 422 and image identifier 444 in the common interface format440, the system 400 may address the request 910 to the appropriatemodule 450 to solicit feature data 420 corresponding to the requestedfeature from the identified portion of the layer.

Furthermore, as described above, in at least some instances, generationof feature data 420 by a module 450 may be based on one or more otherportions of feature data 420 that may be generated by other modules 450.For instance, as illustrated in FIG. 9, a hierarchical model wheremodules 450 operate based on feature data 420 from other modules 450 isshown. For instance, module 1 450 a may rely on Feature 2 and Feature 3in determining Feature 1. As such, module 1 450 a may call module 2 450b to retrieve feature data 420 corresponding to Feature 2 and may callmodule 3 450 c to retrieve feature data 420 corresponding to Feature 3.It may be appreciated that a module 450 may obtain feature data 420 fromfile system 430 and/or another module 450. Module 1 405 a may utilizethe common interchange format 440 by providing a feature identifier 442for Feature 2 and Feature 3 to the respective modules 450 b and 450 calong with an image identifier 444 that corresponds with the image datafor which the respective feature data 420 is to be provided.Furthermore, it may be appreciated that multiple levels of suchdependency may exist such that, for example, determination of Feature 2by module 2 450 b requires feature data 420 corresponding to Feature 4,Feature 5, and Feature 6 to be provided by module 4 450 d, module 5 450e, and module 6 450 f, respectively. In this regard, it may beappreciated that module 2 450 b may similarly utilize the commoninterface format 440 to retrieve the appropriate feature data 420 frommodules 450 d-450 f as shown in FIG. 9.

Accordingly, by use of the common interface format 440, integration ofmodules 450 into the system 400 and intercommunication between modules450 may be improved. For example, rather than having to encodeinstructions for each portion of feature data 420 to be received, upon arequest for feature data 420, the use of the common interface format 440may allow for modular and scalable construction using one or morefeature extraction modules 450. For instance, in the example provided inFIG. 9, absent use of the common interface format 440 for communicationsbetween modules 450, upon the need for feature data 420 corresponding toFeature 1, a user may be required to recognize the need for computationof Features 2-6 by modules 450 b-450 f and encode instructionscorresponding to each modules in addition to the request for Feature 1from module 450 a. This may become particularly burdensome in the casewhere modules 450 are generated using different programming languages,programming paradigms, or the like. However, with use of the commoninterface format 440, each module may be constructed such that it mayautonomously handle external calls for other feature data 420 needed,thus avoiding the need to encode instructions for each operationinvolved in generation of feature data 420. Rather, upon creation of amodule 450, regardless of the manner in which the module 450 is executed(e.g., which programming language, programming paradigm, method ofexecution, etc.), the module 450 may be integrated into the system 400as all requests to the module 450 and responses from the module 450 maybe provided in the common interface format 440 including featureidentifier 442 and image identifier 444 such that intercommunicationwith the module 450 by other modules 450 or between the module 450 andthe file system 410 may be standardized.

Furthermore, it may be appreciated that some feature data 420 may bedetermined on demand, while other feature data 420 may be predetermined,i.e., prior to the feature data 420 actually being requested. Forinstance, certain feature data 420 may be determined only upon request(e.g., by a user or another module 450). Other feature data 420 may beautomatically generated on some occurrence (e.g., receipt of new imagedata 411) and the feature data 420 may be stored for later retrieval.For instance, computationally taxing feature data 420 may bepregenerated such that upon request for the feature data 420 (e.g. by auser or other feature module 420), the feature data 420 may be morequickly retrieved and provided (e.g. from the file system 420). However,in other examples, computationally taxing feature data 420 may be onlycalculated on demand to reduce the computational overhead used. That is,only feature data 420 that is actually requested may be calculated. Asmay be appreciated, which feature data 420 is predetermined ordetermined on demand may be selected by a user of a system or may beprovided in a default state that is changeable by a user.

In any regard, continuing the example shown in FIG. 9, upon receipt ofthe request 910, module 1 450 a may distribute an initialization commandto all feature extraction modules 450 that module 1 450 a depends (i.e.,modules 450 b-450 f in FIG. 9). In this regard, rather than, forexample, sequentially calling each feature module 450 when needed,module 1 450 a may initialize each other feature extraction module 450b-450 f such that those modules may, in parallel, begin to determine orretrieve the feature data 420 relied upon by module 1 450 a. That is, iffeature data 420 for a module 450 is determined on demand, a module 450may begin determining the feature data 420 to be provided to module 1450 a. Additionally or alternatively, if feature data 420 for a module450 is predetermined, the module 450 may access such predeterminedfeature data 450 for the portion of the image for which the feature data420 is requested and provide such feature data 420 to module 1 450 a. Inthis regard, it may be appreciated that the initialization request maycomprise the common interface format 440 including a feature identifier442 and an image identifier 444.

One example of a feature extraction module 450 that may execute in thefeature extraction system 400 described above is a fractal dimensionfeature extraction module. The term “fractal dimension” effectivelydefines the extent to which a lower dimensional function (e.g., a onedimensional line, two dimensional plane, etc.) effectively occupies aspace of a higher dimension (e.g., two dimensional space in the case ofthe one dimension line or three dimensional space in the case of the twodimensional plane). That is, the fractal dimension may provide a measureof the complexity of a function of a given dimension. As an example, twoone dimensional lines defined as

${y = {{{\sin \left( \frac{10}{x} \right)}\mspace{14mu} {and}\mspace{14mu} y} = x}},$

respectively, may have differing fractal dimensions for an interval of−0.2≦x≦0.2. That is, the function

$y = {\sin \left( \frac{10}{x} \right)}$

may have a higher fractal dimension over the interval than the functiony=x as the sine function may occupy much more of the space in theinterval although both functions have a topological dimension of one.

Accordingly, for an image, the gray level values for a given spectralband may be considered as describing a convoluted, two-dimensionalsurface, the fractal dimension of which may provide information aboutthe “roughness” of the two-dimensional surface defined by the gray levelvalues. That is, the image may be conceptualized as a three dimensionalsurface whose height from the normal at each pixel is represented by agray value of the pixel. In this regard, the fractal dimension featureextraction module may be operable to extract feature information inrelation to the fractal dimension of one or more portions of an imagethat may provide useful information regarding the nature of the image.The fractal dimension for a given portion of an image may be determinedusing any appropriate method for calculation of a fractal dimension.

In one particular embodiment of a fractal dimension feature extractionmodule conceptually illustrated in FIG. 6, an image 600 may be dividedinto a plurality of subdivisions 610 of a predetermined size of pixels.This predetermined size of the subdivision 610 of the image 600 may bedefined as N×N pixels. As may be appreciated, the subdivision of theimage may be provided in accordance with a tiling scheme applied togeospatial images such as tiling schemes commonly used in mappingapplications such as, for example, the Google Maps Tiling Scheme,described in the Google Maps API document available from Google, Inc.,Mountain View, Calif., the Bing Maps Tiling Scheme, described in theBing Maps Tile System available from Microsoft Corporation, Redmond,Wash., or a custom tiling scheme that may, for example, be a geographicbased projection. For each predetermined image of size N×N, the imagemay be divided into a plurality of abutting windows 612 within the N×Nimage of size W×W pixels, where W is an integer that complies with theinequalities

$W \leq {\frac{N}{2}\mspace{14mu} {and}\mspace{14mu} W} \geq 3$

and represents a side dimension of the abutting window 612.

As such, for each window 612 of size W×W, a column of boxes 614 having avolume of W×W×W′ may be constructed. In this regard, W′ may beconsidered the height of the column 614. The height may be representedin increments of the side dimension of the abutting window (W). In thisregard, the height may be determined such that the ratio of thepredetermined size of the image (N) to the side dimension of theabutting windows (W) equals the ratio of the maximum pixel gray valuefor a given multispectral band of the image to the height of the column614. That is, where D is the gray level range for the image, the heightof the column 614 may be determined with the equation

$\frac{N}{W} = {\frac{D}{W^{\prime}}.}$

As such, a column of W×W×W′ is created for each abutting window in theimage.

In this regard, for each abutting window 612 of size W×W, the minimumand maximum gray value for each pixel is located within a specific oneof the boxes defined in the W×W×W′ sized column 614. The boxes of sizeW×W×W′ may be numbered with successive integers in ascending orderextending away from the base of the column (e.g., as shown in FIG. 6 asBox 1, Box 2, Box 3 . . . Box n). Once the minimum and maximum grayvalue has been located, the distance separating the minimum and maximumvalues may be computed as n=h−l+1 where h and l are the numbers of boxeswith the highest and lowest values, respectively, over the W×W×W′ sizedbox. This process is repeated for each abutting window 612 of size W×Wto determine a column height W′ and a n value for each. The total numberof boxes to cover the whole image subdivision 610 of size N×N may becalculated as:

$\begin{matrix}{N_{W} = {\sum\limits_{i,j}n}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

This process is further repeated for all values of W that satisfy theabove noted inequalities related to the size of W. In turn, a log-logplot of values of N_(w) versus W may be generated and the negative slopeof the least squares line of the plot may define the fractal dimension.In turn, each subdivision 610 of the image 600 may undergo this processto determine the fractal dimension of each subdivision 610. In turn, afractal map may be generated that is an aggregation of information fromeach unique N×N subdivision 610 of the original image 600. Suchinformation may provide relatively dense textural information per pixel.

Another feature extraction module 450 that may be provided may include atotal variational feature extraction module. In VHR multispectralremotely-sensed Earth imagery, a plurality of spectral bands may becollected. That is, the image sensor used to collect image data may havea plurality of specifically designed sensor portions capable ofdetecting light at a predetermined wavelength. For instance, WorldView 2operated by DigitalGlobe, Inc. collects data in 8 spectral bandsincluding, for example, costal (400-450 nm), blue (450-510 nm), green(510-580 nm), yellow (585-625 nm), red (630-690 nm), red edge (705-745nm), near-infrared 1 (770-895 nm), and near-infrared 2 (860-1040 nm).There may also be a panchromatic sensor capable of detecting black andwhite imagery in a broad wavelength band (e.g., the wavelength band of450-800 nm). Further still, in at least some embodiments, one or morebands in the short-wavelength infrared range (SWIR) may be provided. Forexample, one or more SWIR bands may be provided including, for example,SWIR 1 (1195-1225 nm), SWIR 2 (1550-1590 nm), SWIR 3 (1640-1680 nm),SWIR 4 (1710-1750 nm), SWIR 5 (2145-2185 nm), SWIR 6 (2185-2225 nm),SWIR 7 (2235-2285 nm), and/or SWIR 8 (2295-2365 nm). Other combinationsand/or ranges of SWIR bands generally from about 1195 nm to about 2400nm may be provided in any combination. For any of the foregoing bandsdiscussed, band definitions may be broader and/or narrower than thosedescribed above may be provided without limitation. In any regard, theremay be a plurality of band values corresponding to gray level values(e.g., digital numbers) for each band for each given pixel in a portionof multispectral image data as each band may be representative of theintensity of the surface reflection in a given subset of the totalspectrum.

In any regard, there may be a plurality of band values (e.g., spectralband data 510 shown in FIG. 5) corresponding to gray level values foreach band for each given pixel in the multispectral image data 411.Traditionally, feature sets have utilized combinations of gray levelvalues for each spectral band and/or normalized difference ratios as afeature set for analysis, such as the approach described above relatingto spectral band ratios. However, it may be appreciated that thesevalues may be corrupted by, for example, sensor noise or local textureon a given surface type. Accordingly, prior approaches have beenproposed that filter gray level values over a small spatial window by,for example, taking the mean or median gray level value for a group ofpixels in the spatial window. However, while this type of filtering mayreduce noise, the filtering may blur information across discontinuities,which, for example, may obscure edge detection or other techniques to beapplied to the image. Furthermore, nonlinear median and morphologicalfilters typically operate on a per band basis, where the filter isindependently applied to each of the plurality of spectral bands of animage. This becomes less effective for images having a plurality ofbands.

In turn, the total variational feature extraction module may operate ona plurality of spectral bands of an image (e.g., all 8 spectral bands of8 band image data 411) simultaneously to perform a variationalsimplification across the multiple bands of the image. Accordingly, theoutput of the total variational feature extraction module is a datalayer comprising a filtered set of image bands (i.e., a filtered image)that may be utilized in further image processing (e.g., utilizing thefiltered gray level values or normalized ratios therefrom).

The total variational feature extraction module may be a globaloptimization that all image pixels from a plurality of spectral bandsare optimized jointly using the formula:

ƒ(u)=τ_(i) |u _(i) −x _(i)|+λ|grad(u _(i))|  Equation 3

where i represents a multivalued vector at a pixel, u is the filteredimage stack (row, column, band), and x is the input image. The L1 vectornorm of a given function g(x) is represented by the value |g(x)| in theabove equation, where the L1 vector norm is defined by the equation:

$\begin{matrix}{{{(x)}} = \sqrt{\sum\limits_{i = 1}^{n}x_{i}^{2}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

The function grad( ) represents a vector-valued spatial gradient definedas:

|grad(u _(i))|=√{square root over (|gradx(u _(i))|²+|grady(u_(i))²|)}{square root over (|gradx(u _(i))|²+|grady(u_(i))²|)}  Equation 5

where gradx( ) and grady( ) represent the spatial gradient in the imagex and y directions, respectively. In this regard, Equation 3 may beoptimized to determine a global minimum for the image over a pluralityof bands. As such, the best filtered image u is found when Equation 3 isglobally minimized over the entire image including all spectral bands.In this regard, it may be appreciated that the first term of Equation 3may penalize the filtered image as the filtered image differs from theoriginal image. The second term of Equation 3 attempts to regularize thefiltered image u such that the resulting image values of u are spatiallyhomogenous. To this end, it may be appreciated that as the L1 norm isutilized (e.g., in contrast to the L2 norm), in the first term, anoccasional large discrepancy or outlier may be tolerated in theminimization. Similarly, the use of the L1 norm in the second termallows for abrupt edge discontinuities in the second term. Additionally,in the second term A represents a smoothing factor that allows fortuning of the desired smoothness in the resulting filtered image u.

In this regard, Equation 3 may be optimized to determine the resultingglobal minimum for the function ƒ(u). The solution may be obtained usingany optimization approach, however, it has been found that in apreferred embodiment, a Split Bregman technique may be employed.Additionally, Gauss-Seidel updates may be performed. A discussion of theSplit Bregman technique is disclosed in Goldstein, Tom and Osher,Stanley, The Split Breqman Method for L1 Regularized Problems, availableat ftp://ftp.math.ucla.edu/pub/camreport/cam08-29.pdf, the entirety ofwhich is incorporated by reference herein. It has been found that thisapproach to the optimization is quite fast and maps well to execution ona GPU (e.g., a CUDA processor) due to the highly parallel nature of thealgorithm.

Another feature extraction module 450 that may be provided is arotational invariant histogram of gradients module. In this regard, oneskilled in the art will recognize that histograms of oriented gradients(HOGs) is an established technique for computing and histogramminggradients in images based on the orientation and magnitudes of thegradients over some window. For example, Dalal, Navneet and Triggs,Bill, Histograms of Oriented Gradients for Human Detection,International Conference on Computer Vision and Pattern Recognition,(2005), discusses one such approach and is incorporated by reference inits entirety.

However, previous approaches to HOGs relied on underlying assumptionsregarding the orientation of objects to be identified using HOGs. Forinstance, prior work may have assumed that human figures would generallyappear vertically in a standing position. However, in the case of VHRmultispectral remotely-sensed Earth imagery, such assumptions based onsuspected orientations may be of very little to no value as theorientation of objects in such imagery may be provided over a pluralityof orientations or at random. As such, a rotationally invariant approachto HOGs is proposed and discussed herein.

Specifically, as may be appreciated, a gradient for a specific pixel mayinclude an x (i.e., horizontal) and y (i.e., vertical) component thatmay be orthogonal to each other. With reference to FIG. 7, in arotationally invariant approach to HOG, an image 700 may be representedin gradient domain 750. In the gradient domain, 750, N annular bins 752may be established in addition to a low gradient region disposed in thecenter 754 of the N annular bins 752. In this regard, each annular bin752 may correspond with a portion of a ring surrounding the center 754.FIG. 7 depicts an example of one arrangement of the annular bins 752 andcenter 754 as arranged in a plurality of cells 750 where there are 8annular bins 752 and a single center 754. In another embodiment, asecond gradient magnitude threshold may be defined to yield 2N+1 bins752 (i.e., 16 annular bins 752 and one center bin 754). Each pixel froman image may be categorized into a bin 752 based on the direction and/orvalue of a gradient at the pixel as represented in the gradient domain750.

Additionally, as shown in FIGS. 8A and 8B, attached to a pixel may be aset of P pooling windows 860 with set relative offsets from the pixel.Also, a larger window 862, G, may be defined over which the gradientsare to be histogrammed. The angular bin N_(i) from the gradient domain750 with the largest histogram value (from FIG. 7) is selected as theprimary direction. That is, for the example shown in FIG. 7, one of theannular bins 752 is chosen based on the chosen angular bin having thelargest histogram value. For each of the pooling windows 860, 2N angularbins may be stored to provide proper oversampling.

All P pooling windows 860 may be tied to the primary direction of thewindow 862 as determined from the selected angular bin 752 with thelargest histogram value. Accordingly, shifting the pooling windows 860to correspond to the primary direction results in only one degree offreedom is being lost over all windows when achieving rotationalinvariance. Once the primary direction is determined from selecting theangular bin 752 with the largest magnitude of histogram, the window 862is also used to rotate the configuration compared to the primarydirection. Accordingly, the pooling windows 860 shift position and theinternal orientation of the pooling windows 860 may be rotated, as shownin FIG. 8B.

In an embodiment, integral images may be used to allow all poolingwindows 860 to be calculated to generate a HOG for each pooling window860. The internal orientation of the pooling windows 860 is rotated byshifting the resulting histogram by a fixed number of angular bins 852determined for the window G 862.

Accordingly, this approach to HOG may provide for rotational invariance.As such, regardless of the orientation of the object to be identified,the HOG produced may be capable of producing useful features associatedwith the identification of such features. As may be appreciated, for VHRmultispectral remotely-sensed Earth imagery, such rotational invariancemay be particularly useful given the lack of uniform orientation offeatures to be identified.

Yet another feature extraction module 450 that may be provided is aGabor wavelet feature extraction module with rotational invariance. Inthis regard, while Gabor filters have been utilized in edge detection ofimages in image processing, the Gabor wavelet feature extraction moduledescribed below may provide for rotational invariance, which asdescribed above may be particularly beneficial in the context of VHRmultispectral remotely-sensed Earth imagery.

In this regard, the image may be convolved with a Gabor waveletfunctions as follows. For the real portion of the function, thefollowing equation may be provided:

$\begin{matrix}{{\left( {x,{y;\lambda},\theta,\psi,\sigma,\gamma} \right)} = {{\exp \left( {- \frac{x^{\prime 2} + {\gamma^{2}y^{\prime 2}}}{2\; \sigma^{2}}} \right)}{\cos \left( {{2\; \pi \frac{x^{\prime}}{\lambda}} + \psi} \right)}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

Furthermore, the imaginary component of the function may be representedby the function:

$\begin{matrix}{{\left( {x,{y;\lambda},\theta,\psi,\sigma,\gamma} \right)} = {{\exp \left( {- \frac{x^{\prime 2} + {\gamma^{2}y^{\prime 2}}}{2\; \sigma^{2}}} \right)}\mspace{11mu} \sin \mspace{11mu} \left( {{2\; \pi \frac{x^{\prime}}{\lambda}} + \psi} \right)}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

In both the functions provided above, the values of x′ and y′ may beprovided as follows:

x′=x cos θ+y sin θ  Equation 8

y′=−x sin θ+y cos θ  Equation 9

In the foregoing equations, λ represents the wavelength of thesinusoidal factor of the filter. The value of θ represents theorientation of the normal to the parallel stripes of the Gabor function.The value of ψ is the phase offset, σ is the sigma of the Gaussianenvelope, and γ is the spatial aspect ratio, which specifies theellipticity of the support of the Gabor function.

Using the foregoing definitions of the Gabor wavelet functions, acalculation using the module is calculated for each pixel usingdifferent values of θ. For instance, a plurality of orientations may beprovided for the calculating including, for example, varying values at22.5 degree increments (e.g., 0 degrees, 22.5 degrees, 45 degrees, 67.5degrees, 90 degrees, 112.5 degrees, 135 degrees, 157.5 degrees, and 180degrees). Other intervals may be utilized such as 11.25 degreeincrements, 45 degree increments, or other appropriate angularincrements. In an embodiment, the calculations made for the variousvalues of θ are made keeping the other variables for the Gabor filterconstant. In other embodiments, the calculations may be made at eachangular increment using different values for the other variables in theequation to establish a plurality of scales each calculated over thevarious intervals of θ as described above.

In any regard, once the values for each angular increment for each pixelhave been calculated, the results of the calculations for the variousintervals may be further processed to achieve rotational invariance. Forexample, in an embodiment, the sum of the absolute values of the resultsfor all orientations at a given pixel may be calculated. That is, thesum of all calculations for each value of θ may be provided. In anotherembodiment, the sum of the magnitude for the values for themultiplication of convolution values for the Gabor functions that areperpendicular may be provided. That is, for example, the results of thecalculation for the orientation (i.e., a θ value) of 0 degrees may besummed with the result of the calculation for the orientation of 90degrees, the results of the calculation for the orientation of 22.5degrees may be summed with the result of the calculation for theorientation of 112.5 degrees, and so on. Further still, the differencebetween the maximum and minimum values of the magnitudes of thecalculations at the various orientations may be calculated for eachpixel

In this regard, the Gabor wavelet feature extraction module may outputany of the foregoing values based on the calculations made at eachorientation. In turn, the output may be rotationally invariant as theorientations calculated may account for rotations of features in the VHRremotely-sensed multispectral Earth images examined. In this regard, thevarious values described above calculated based on the results of theGabor filter over the various orientation values may result in arotationally invariant output for each pixel. As described above, suchrotational invariance may be particularly useful in the context ofremotely-sensed Earth images.

Another feature extraction module 450 that may be provided is aclustering histogram feature extraction module. The clustering histogramfeature extraction module may include computing a histogram of clustervalues in a predefined area surrounding a pixel. For example, each pixelof an image may be classified into a predefined set of classes. Theclasses may be established and the pixels may be categorized accordingto any appropriate technique. For example, in an embodiment a clusteringtechnique such as k-means clustering using Euclidean or Mahalanobisdistance may be used. In one particular embodiment for VHRremotely-sensed multispectral Earth images, the classes into whichpixels are categorized may correspond to land classifications e.g.including land classes or land use classes defined for the image.

In any regard, once the pixels of the image are classified, for eachpixel a histogram of cluster values for all pixels within a predefineddistance of the pixel is created. The histogram may be built by countingthe number of pixels within the predefined distance from the subjectpixel that belong to each particular class. In this regard, thehistogram may provide details regarding the nature of the surroundingpixels to any given pixel. In this regard, the histograms for the pixelsof an image may provide useful information that may be particularlysuited to the analysis of VHR remotely-sensed multispectral Earthimagery.

For instance, in the context of classification of land use or in objectrecognition, a feature that reflects the surrounding neighborhood ofpixels may provide useful insight. For instance, it may be recognizedthat certain conditions may be recognized from the surrounding pixelswhen classifying the pixel or identifying an object. As an example, adock may be surrounded by water such that a pixel belonging to a dockmay reflect in the histogram created by the clustering histogram featureextraction module that neighboring pixels include those classified aswater. As such, when identifying the dock or classifying the pixel, itmay be recognized that a dock is generally surrounded by water. As such,the surrounding pixels may provide insight into properties of a givenpixel, and such surrounding pixel information may be captured in thehistogram created. Other examples may also be provided such as in anapplication of land use classification where, for example, the histogramfor a number of surrounding pixels may provide insight into the type ofland use (e.g., a mix of roads, roof and vegetation may indicate aresidential area whereas a lack of vegetation may indicate a commercialarea). In this regard, the clustering histogram feature extractionmodule may provide useful information that may be leveraged in furtherimage analysis.

Another feature extraction module 450 that may be provided is agray-level co-occurrence matrix (GLCM) module. The GLCM is a tabulationof how often different combinations of pixel gray level values occur inan image. As such, the GLCM may provide textural information regardingan image that may be useful in classification techniques. A fulldescription of an embodiment of a GLCM module may be found inHall-Beyer, The GLCM Tutorial Home Page, Version 2.10 available athttp://www.fp.ucalgary.ca/mhallbey/tutorial.htm, the entirety of whichis incorporated herein by reference.

In turn, once a GLCM has been generated by the GLCM module, statisticsmay be generated based on the GLCM. For example, mean, variance, andcorrelation calculations may be performed. Furthermore, contrast groupsmay be calculated that are related to contrast use weights related tothe distance from the GLCM diagonal. Examples include contrastcalculations or “sum of squares variance” for the GLCM, dissimilaritycalculations, or homogeneity calculations or “inverse differencemoments”. Additionally, orderliness groups may be calculated such asangular second moment calculations, maximum probability calculations, orentropy equations.

A number of parameters may be established for creation and use of GLCMsin the GLCM module. For instance, the size of the window over which aGLCM is calculated may be provided as a variable definable by a user. Inthis regard, the window is preferably large enough to cover features tobe identified, but small enough to be localized. For example,calculation of the GLCM in a forest must at least have a window sizelarge enough to identify a tree or calculation of the GLCM in anagricultural field may have a window size roughly correlated to the rowspacing of crops planted in the field. Furthermore, the direction of theoffset of the GLCM may be definable as may be the bands for which theGLCM is calculated and the measures (e.g., those described above)calculated for the GLCM.

Yet further feature extraction modules 450 may be provided withoutlimitation. An example of a further feature extraction technique thatmay be utilized include a mean feature approach. In this approach, themean values for all bands based on a defined kernel are extracted. Thekernel may be sized, for example, to be a 3×3 pixel kernel althoughother values may be used. In this regard, the mean feature module mayprovide smoothing over the kernel.

Another feature extraction module 450 that may be used includes adigital elevation model (DEM) module that may analyze image data inrelation to a DEM. In this regard, some land classes may be tied torelatively flat areas (e.g., crops, wetlands, etc.). A such, a modulemay generate features based on a DEM corresponding to the geographycovered by the image. In addition to providing useful informationdirectly related to land classes, reference to DEM data for a portion ofthe image may influence illumination of various portions of the imagethat may be accounted for in processing (e.g., identification of shadyareas based on elevation profiles).

As may be appreciated, the foregoing feature extraction models 450 maygenerate data layers for the feature stack 500 that may each beparticularly suited for various analysis tasks carried out relative tothe image data 411. In this regard, a user may be capable of selectingone or more of the feature extraction modules 450 for execution withrespect to image data 411 (e.g., including the identity or order of themodules 450). Further still, at least one parameter related to theexecution of a feature extraction module 450 may be adjustable by a userto tailor the feature extraction module 450 to a particular application.In this regard, the selection of certain ones of the feature extractionmodules 450 and/or customization of parameters related thereto mayfacilitate improved processing efficiency as unused and/or unhelpfulfeature data layers may not be generated based on a user's selection ofa limited set of the feature extraction modules 450.

Furthermore, the feature modules 450 may be automatically executed aspart of an analysis process. For instance, the data store 410 may beaccessible by an image processing module 460 that may be operable toexecute analysis with respect to image data 411 and/or feature data 420.Examples of image processing modules 460 may include land use/landclassification modules, data mining modules, indexing modules, objectdetection modules, change detection modules, parameter monitoringmodules, or any other appropriate image analysis module that may analyzeimage data 411 and/or feature data 420. In any regard, the imageprocessing module 460 may provide instructions regarding the identityand/or order of application of feature extraction modules 450 to theimage data 411. For example, the image processing module 460 may providea request in the interface format 440. Thus, the feature extractionsystem 400 may autonomously and automatically, in response to therequest from the image processing module 460, operate feature extractionmodules 450 on image data 411 to generate feature data 420. In thisregard, the system 400 may determine a subset of the plurality offeature extraction modules 450 to be executed in response to a requestby an image processing module 460. Furthermore, a sequence or order ofoperation of feature extraction modules 450 may be determined.Furthermore and as described above, the feature extraction modules 450may be at least partially hierarchical such that a first featureextraction module 450 may generate feature data 420 using feature data420 generated by another of the feature extraction modules 450. As such,the system 400 may operate in at least a partially autonomous orautomatic manner to generate feature data 420 for use in image analysisby the image processing module 460.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and description isto be considered as exemplary and not restrictive in character. Forexample, certain embodiments described hereinabove may be combinablewith other described embodiments and/or arranged in other ways (e.g.,process elements may be performed in other sequences). Accordingly, itshould be understood that only the preferred embodiment and variantsthereof have been shown and described and that all changes andmodifications that come within the spirit of the invention are desiredto be protected.

1. A method for extracting feature data from very high resolution (VHR)remotely-sensed multispectral Earth imagery, comprising: storing VHRremotely-sensed multispectral Earth image data in a file system;providing a plurality of feature extraction modules, wherein eachrespective feature extraction module is operable to analyze the imagedata to generate feature data regarding the image data; accessing theVHR remotely-sensed multispectral Earth image data in the file systemwith one or more of the plurality of feature extraction modules, whereinthe one or more feature extraction modules each access the file systemusing a common interface format; executing the one or more featureextraction modules to generate feature data corresponding to the VHRremotely-sensed multispectral Earth image data; and receiving thefeature data at the file system from the one or more feature extractionmodules, wherein the feature data is received from each of the one ormore feature extraction modules using the common interface format. 2.The method of claim 1, wherein the VHR remotely-sensed multispectralEarth image data comprises spectral band data corresponding to at least8 multispectral bands.
 3. The method of claim 2, wherein themultispectral bands collectively range from at least about 300nanometers in wavelength to at least about 1100 nanometers inwavelength.
 4. The method of claim 2, wherein the image data furthercomprises a plurality of short-wavelength infrared (SWIR) bands.
 5. Themethod of claim 4, wherein the plurality of SWIR bands range from atleast about 1100 nanometers in wavelength to at least about 2400nanometers in wavelength.
 6. The method of claim 1, wherein the featureinformation received from the one or more feature extraction modules isstored in corresponding relation to the portion of the VHRremotely-sensed multispectral Earth image data from which the featuredata was generated.
 7. The method of claim 1, wherein at least two ofthe plurality of feature extraction modules comprises differentprogramming languages.
 8. The method of claim 1, wherein the pluralityof feature extraction modules are independently developed and eachcommunicate with the file system using the common interface format. 9.The method of claim 1, wherein the common interface format comprises afeature identifier and an image identifier.
 10. The method of claim 9,wherein the feature identifier corresponds with particular feature datagenerated by a given feature extraction module.
 11. The method of claim10, wherein the image identifier comprises an indication of a portion ofthe image for which feature data is to be generated.
 12. The method ofclaim 11, further comprising: determining a subset of the plurality offeature extraction modules required to generate feature data, whereinfirst feature data generated from the execution of a first featureextraction module is used during the executing of a second featureextraction module to generate second feature data.
 13. The method ofclaim 12, wherein the determining further comprises ordering theexecution of the subset of feature extraction modules.
 14. The method ofclaim 12, further comprising: distributing an initialization command toeach of the subset of the plurality of feature extraction modules,wherein the initialization command comprises a feature identifier and animage identifier according to the common interface format.
 15. Themethod of claim 1, wherein at least one of the feature extractionmodules generates feature data prior to receiving a request for thefeature data from an image processing module.
 16. The method of claim 1,wherein at least one of the feature extraction modules generates featuredata on demand in response to a request for the feature data from animage processing module.
 17. The method of claim 13, wherein thedetermining is at least partially based on a feature of interest to beidentified from the image data as requested from an imaging processingmodule. 18.-23. (canceled)
 24. The method of claim 1, wherein at leastone of the feature extraction modules comprises a total variational (TV)feature extraction module.
 25. The method of claim 24, wherein the TVfeature extraction module comprises a multispectral filtering moduleusing a plurality of spectral bands of an image to generate a filteredimage corresponding to the image.
 26. The method of claim 25, whereinthe number of spectral bands used in the filtering operation comprisesat least 8 multispectral bands.
 27. The method of claim 26, wherein thefiltering operation comprises minimizing a function including at least afirst term representing the difference between the image and thefiltered image and at least a second term representing the spatialhomogeneity of the filtered image.
 28. The method of claim 27, whereinthe first term and the second term include a multivalued vector for eachpixel representative of the image or filtered image, respectively,wherein the multivalued vector values include a component for each row,column, and multispectral band in the image.
 29. The method of claim 28,wherein at least one of the first term or the second term include an L1norm.
 30. The method of claim 29, wherein the second term is weighted bya smoothing factor.
 31. The method of claim 30, wherein the minimizationof the function is solved using a Split Bregman technique.
 32. Themethod of claim 31, wherein the Split Bregman technique includesGauss-Seidel updates.
 33. The method of claim 32, wherein theminimization is solved by execution on a graphics processing unit (GPU).34.-38. (canceled)
 39. The method of claim 1, wherein at least one ofthe feature extraction modules comprises a Gabor wavelet featureextraction module.
 40. The method of claim 39, wherein the Gabor waveletfeature extraction modules is operative to generate feature data by:applying a plurality of Gabor wavelet functions to an image, whereineach of the plurality of Gabor wavelet function includes a differentorientation; and computing at least one feature based on at least two ofthe plurality of Gabor wavelet functions.
 41. The method of claim 40,wherein the at least one feature comprises at least one of a sum of theabsolute values of a first Gabor wavelet function of the plurality ofGabor wavelet functions and a second Gabor wavelet function of theplurality of Gabor wavelet functions, a sum of the magnitude of valuesof the product of each perpendicularly oriented Gabor wavelet functionof the plurality of Gabor wavelet functions, and the difference of themaximum and minimum values of magnitude of all of the plurality of Gaborwavelet functions for all image data.
 42. The method of claim 41,wherein for each orientation of the plurality of Gabor waveletfunctions, a plurality of different Gabor wavelet functions with adifferent scale are calculated.
 43. The method of claim 42, wherein thedifferent scales comprise varied values for at least one of a wavelengthof the Gabor wavelet function or an envelope of the Gabor waveletfunction.
 44. The method of claim 43, wherein each of the plurality ofGabor wavelet functions are applied to each pixel of each multispectralband of an image comprising a plurality of multispectral bands.
 45. Themethod of claim 44, wherein the at least one feature is determinedindependently for each of the plurality of bands.
 46. The method ofclaim 45, wherein the at least one feature is determined with respect toall of the plurality of bands. 47.-51. (canceled)
 52. A system forgenerating feature data from very high resolution (VHR) remotely-sensedmultispectral Earth imagery, comprising: a data store comprising VHRremotely-sensed multispectral Earth image data in a file system; aplurality of feature extraction modules, wherein each respective featureextraction module is operable to analyze the image data to generatefeature data regarding the image data; wherein the one or more featureextraction modules each access the file system using a common interfaceformat to access the image data and store feature data. 53.-67.(canceled)
 68. A method for generating feature data from image datausing a plurality of hierarchical feature extraction modules,comprising: providing a plurality of feature extraction modules, whereineach respective feature extraction module is operable to generatecorresponding feature data for image data, wherein the plurality offeature extraction modules comprises at least a first feature extractionmodules operable to generate first feature data and a second featureextraction module operable to generate second feature data; firstgenerating the first feature data with the first feature extractionmodule; communicating the first feature data to the second featureextraction module using a common interface format; and second generatingthe second feature data at least in part based on the first featuredata.